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The DZERO experiment located at Fermilab has recently started Runll with an upgraded detector. The Runll 
physics program requires the Data Acquisition to readout the detector at a rate of 1 KHz. Events fragments, 
totaling 250 KB, are readout from approximately 60 front end crates and sent to a particular farm node for Level 
3 Trigger processing. A scalable system, capable of complex event routing, has been designed and implemented 
based on commodity components: VMIC 7750 Single Board Computers for readout, a Cisco 6509 switch for 
data flow, and close to 100 Linux-based PCs for high-level event filtering. 



1. Introduction 

The Level 3 Data Acquistion System (L3DAQ) [1, 2] 
transports detector component data located in VME 
readout crates to the processing nodes of the Level 3 
trigger filtering farm. At a rate of 1kHz, sixty-three 
VME crates must be read out for each event, each 
containing 1-20 kB of data distributed among VME 
modules. The total event size is approximately 250 
kilobytes. 

As shown in figure 1, the system is built around 
a single Cisco 6509 [3] ethernet switch. A schematic 
of the communication and data flow in the system is 
shown in figure 2. All nodes in the system are based 
on commodity computers and run the Linux operating 
system. TCP /IP sockets implemented via the ACE [4] 
CH — h network and utility library are used for all com- 
munication and data transfers. 



2. Operation 

The Supervisor process provides the interface be- 
tween the main DO run control (COOR) and the 
L3DAQ system. When a new run is configured, the 
Supervisor passes run and general trigger information 
to the RM and passes the COOR-provided L3 filter 
configuration to the IO /EVB process on relevant farm 
nodes, where it is cached and passed on to the Level 
3 filter processes. 

A single-board computer (SBC) in each VME crate 
reads out the VME modules and sends the data to 
one or more farm nodes specified by routing instruc- 
tions received from the Routing Master (RM) process. 
An Event Builder (EVB) process on each farm node 



builds a complete event from the received event frag- 
ments and makes it available to Level 3 trigger filter 
processes. 

The SBCs are single-board computers with dual 
100 Mb/s Ethernet interfaces and a VME-to-PCI in- 
terface. An expansion slot is occupied by a digital-I/O 
(DIO) module, used to coordinate the readout of VME 
modules over the VME user (J3) backplane. A custom 
kernel driver on the SBC handles interrupt requests 
from the DIO module which are triggered by readout 
requests from the crate-specific electronics. On each 
readout request the kernel module performs the VME 
data transfers and stores the event fragment in one of 
several buffers in kernel-memory. 

A user-level process on the SBC receives route in- 
formation from the Routing Master in the form of 
Route Tags that contain a unique event identifier (L3 
transfer number) and the indices of the farm nodes to 
which that event should be sent. If the Route Tag's L3 
transfer number matches that of the transfer number 
embedded within the head event fragment in the ker- 
nel buffers, the event fragment is sent to the specified 
farm nodes. 

An Event Builder process (EVB) on each farm node 
collates the event fragments received from SBCs into 
complete events, keyed by L3 transfer number. For 
each event the EVB receives an expected-crate list 
from the RM in order to determine when an event 
is complete. Complete events are placed in shared 
memory event buffers for processing by the Level 3 
filter processes. The EVB routinely informs the RM 
of the number of available event buffers that it has, 
so that the RM will not route an event to a farmnode 
unless the event can be processed immediately. 

The Routing Master program executes on an SBC 
in a special VME crate which contains a hardware 
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interface to the DO trigger framework (TFW). The 
TFW provides trigger information and the L3 trans- 
fer number for each event and allows the RM to asyn- 
chronously disable the firing of triggers. For each 
event the RM program chooses a node for processing 
based on the run configuration, the trigger informa- 
tion, and the number of available buffers in the set of 
nodes configured to process the type of event. A node 
is chosen in a round-robin fashion from amongst the 
set of nodes with the most free buffers. If the number 
of available free buffers is too low, the RM instructs 
the TFW to disable triggers so that the farm nodes 
have time to catch up. 

To avoid dropped packets in the main switch, data 
flow is limited by setting the TCP/IP receive win- 
dow size on the farmnodes. The window size is set 
such that the product of the number of connection 
sources and the receive window size is smaller than 
the switch's per-port output buffer memory. 

3. Conclusion 

The DZERO Level 3 data acquistion system has 
been built from commercially available hardware: sin- 



gle board VME computers, ethcrnct switches, and 
PCs. The software components rely upon high level 
programming languages; the Linux operating system; 
widely-used, open libraries; and standard networking 
protocols. The system has performed reliably since 
commissioning in May 2002. Additional details are 
availble from the references [2]. 
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Figure 1: The physical network configuration of the L3DAQ system. 
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Figure 2: Schematic illustration of the information and dataflow through the L3DAQ system. 
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